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PROCESS FOR MEASURING A THREE-DIMENSIONAL OBJECT OR A 



The invention is a process for determining the 
geometry, position and orientation of one or several 
objects in an environment. The objective is to provide 
dimensional or measurement information on three- 
dimensional primitives (dots, straight lines, circles, 
cylinders, .etc . ) ( representing these objects . .using .-a 
projection , on images acquired by, one cor several 
sensors. These ^ dimensional data are used:.: -fori i the 
dimensional check of manufactured objects (prototype or 
series production) , the measurement of structural 
deformation, and modeling of industrial environments.. 

There are several major families of processes to 
accomplish this type of measurement. Some involve 
direct measurement of objects in the environment by the 
tip of a feeler, but this method cannot always be 
applied and it becomes very long as soon as the 
environment becomes voluminous or cluttered, or if its 
shape is complicated; This method is unthinkable when 
the environment is the size of a complete room. Other 
methods make use of range finding, in other words 
distances are measured to various dots in the 
environment without any physical contact; a laser is 
moved towards these dots, one at a time, and the 
measurement is made on the flight time or phase shift 
of the wave. A mirror or a mechanical system is used 
to continuously move the laser ray towards other dots, 
to enable fast measurements of the environment, but it 



SET OF OBJECTS 



DESCRIPTION 



2 

is found that this method is not always very precise 
(although the resolution is limited only by the laser 
scanning system) and is accompanied by errors when the 
beam touches reflecting objects, and it is also 
5 necessary to maintain the stability of the mechanical 
system while scanning and to take care to guarantee the 
safety of any persons within the measurement volume. 

Other methods include optical methods in which a 
camera is moved in front of the environment to be 

10 measured and takes a sequence of images. The details 
of the environment ! "are identified on the. different 
images:..: and : their. -, position is - calculated:-.^' - by 
triangulation .based*, on their position on the ...dif f erent 
images and the known positions as the camera ; advances , 

15 as a function of image taking parameters of the camera. 
Sometimes, a network of dots is identified in the 
environment, these dots being illuminated by a laser ..or 
projector in a beam of rays/ additional light may be 
added to better illuminate the surroundings around the 

2 0 dots network and to make it stand out from the rest of 
the environment. The use of an illumination means 
resembles range finding processes and introduces 
corresponding disadvantages of inaccuracy and lack of 
safety, that do not always compensate for the speed and 

25 ease of identification and the calculations that can 
frequently be carried out. 

In other methods, the dots to be measured are 
light sources, reflecting or colored marks previously 
placed in the environment . These methods give good 

30 results if the marks and their positions are suitably 
chosen, but they are not applicable in all cases and 
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particularly for large complicated environments; in 
particular, they are useful for monitoring the position 
of a determined object moving in the environment, 
rather than for measuring the environment itself. 
5 Finally, other optical processes are based on the 

lack of marks in the environment and on measuring some 
points of interest in images. The points of interest 
are chosen automatically, or the operator may choose 
them in the case of interactive processes. Interactive 

10 processes have the advantage thatv.they are universal or 
theoretically, /-applicable to any .environment -but . it is 
•not always ~ easy to have a sufficiently: .-larger number of 
points of v -interest that are common to allrimages; the 
step in 'which each dot is , identified . on different 

15 images may be difficult; furthermore, a description of 
an environment by even a large number of dots is not 

very .meaningful . 

The invention consists of a process included in 
purely optical methods and more precisely methods that 

20 include an image analysis. This new process may 
include triangulation, but it is fundamentally 
different from previous processes, in that in this case 
we are interested in distinguishing details of the 
environment rather than drawing up a map of the dots in 

25 it. It is often easier and more useful to discern a 
specific element of the environment and to distinguish 
it from the rest, rather than to have a complete but 
indistinct knowledge about this environment. In the 
frequently encountered case of measuring industrial 

30 rooms, this consideration will be particularly 
important when there are a lot of different elements 
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and obstacles of a simple shape, that are superposed 
and create a very complicated relief, but 
interpretation of the resulting representation is much 
easier when these elements are distinguished and when 
5 they are characterized by a few position and shape 
parameters . 

The process has many advantages: there is not 
really any dot in making specific marks in the 
environment; a much larger portion of the information 

10:: ' in the - images is used than if points of ^interest only 
are : considered, which must give •betters v preeision of the 
•::r. . resul^ting ".-.representation; ;. the::;?process :i*s efficient 
evenl'v.with:?. a variety of dif f using.:^:or reflecting 
surfaces; it is applicable to a variety of volumes, 

15 possibly very large; the acquisition time is very 
fast,, a : f ew tens of milliseconds; the ^process may be 
fully automated; the representation may be completed 
later by adding new entities which had been neglected 
earlier, or by correcting it with updates or other 

20 operations; and since it immediately supplies a 

correct model of the environment, it can be used 
immediately, whereas a map of dots needs to be 
interpreted. 

The process is based on a system composed of five 
25 main modules defined in the following list: 

- an image processing module that precisely 
locates natural contours of objects to be 
reconstructed; 

- a reconstruction and positioning module that 
30 determines the geometric parameters of objects 

and the situation of the image capture system; 
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- an identification module that automatically 
searches for natural contours of previously 
reconstructed objects; 

- a module matching points of interest to help 
5 replace these contours of reconstructed objects 

on a new image; 

- and a reconstruction module in blocks making an 
overall (summary) calculation based on all 
available information and very much improving 

10 ' * • ; c .vthe precision . * * .v ; 

• :-;K.alFhe use of this process. ..requires one or several. 

•j. ^T^prie^yi'OHsly calibrated video:^^ cameras (although 
i: calibration is not necessary .if Ydofclfeype primitives are 
- used * exclusively) , in order to\ determine the relation 

15 between any dot on the image and the position of the 
associated light ray. Preliminary .calibrations have 

already been . described by different authors, for 

example the article by Viala, Chevillot, Guerin and 
Lavest : "Mise en oeuvre d'un procede d' eta.lonna.ge 

2 0 precis de camera CCD - Implementation of a process for 
precise calibration of a CCD camera" presented at the 
11 th Conference on Shape Recognition and Artificial 
Intelligence (Clermont-Ferrand, January 20 to 22, 
1998) . When several cameras are used, the system is 

25 said to be stereoscopic and is capable of automatically 
giving a three-dimensional model of the environment by 
searching for corresponding dots on the images and 
triangulation . If a single camera is used, the same 
result can be obtained by successive images by moving 

30 the camera by a determined distance. This distance may 
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also be determined afterwards by calculation, if a 
standard meter is available in the environment. 

In summary, the invention relates to a process for 
measuring three-dimensional objects in a three- 
5 dimensional environment, consisting of taking at least 
one image by at least one camera and creating a 
representation of the environment based on an analysis 
of the image, characterized in that the analysis 
comprises detection of discontinuities in the 

10 .appearance of the image, v. : a' combination: of ~. 
^discontinuities detected., at 1 geometric contours defined.. 

,; 'i.ci!.:: r oii"' the image by parameters> v c.ant^adj ustment of ■ contours.!:; :■: .. -■• ^.^r:/:^ 

"VTi.dy-^t'O' discontinuities by varying. the parameters, an :i :\^yt:.^$c? 

. * estimate of the shape and- position in the environment • - 

15. of geometric objects projecting -onto the image ii.*: 
: ' according to the said contours ,: the representation : 
showing the said objects. 

The representation of the environment is added to 
every time that a new image is taken or when additional 

20 information is supplied. The process can also include 
initial estimates of the position of objects or the 
camera starting from information given manually or in a 
computer description file. 

In general, the process can be carried out with 

25 many alternatives and with flexibility depending on the 
situation encountered. One possibility with some of 
the best embodiments is a correction to the position of 
objects by estimating positions of projections of the 
objects onto the images, based on the respective 

3 0 positions of the camera after the images have been 
taken, and by adjusting the estimated positions of the 
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projection based on the measured positions of the 
projection on the images. 

This correction is usually made during a final 
summary calculation in which the total representation 
error is estimated and then minimized; the estimate of 
camera parameters can also be corrected. 

We will now describe a specific embodiment of the 
invention with reference to the following figures: 

• Figure 1A is a diagram showing an examination 
system in a typical environment; 

• Figure IB ' illustrates how the environment. 1 vis: 
perceived on 'an : image ;" 

• Figure 2 diagrammatically shows processing 
modules of the examination system; 

Figures 3, 4, 5 and 6 illustrate contour models ; 

• and figures 7 and 8 describe some notations used 
in the description. 

The modules mentioned above are referenced with 
marks 20 to 24 on figure 2; they will now be described 
in sequence using the example in figure 1A, in which 
the environment to be recognized comprises a pipe 1 
with a double bend at 2 and 3, and finishing at an 
opening 4, a box 5 and an lamp 6. The display 
equipment, called sensor 7, comprises two rigidly 
installed video cameras 8 and 9 (although they could be 
adjustable if necessary) on a common support 10 
connected to an operating system 11 that in particular 
comprises a memory 12 in which the images from cameras 
8 and 9 are stored, and a processing unit 13. The 
process according to the invention consists of using 
the images one after the other to create a 
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representation of the environment that is added to and 
clarified when interpreting each new image. This work 
is essentially automatic, although in some 
circumstances an operator must apply his judgment in 
practice to complete or correct the representation. 

A representation of the environment means a 
measurement of geometric or dimensional characteristics 
of one or several objects, measurement of geometric or 
dimensional characteristics of elements or objects 
forming a scene, or an environment. This term also 
relates. - to 7 the. measurement of the . position T.-vand 
orientation of one. or several objects, i - o. *<. 

A camera 1 image .consists of a network of dots.: with 
different shades of gray, that are converted into 
digital values to be , stored in memory 12 . Figure IB 
shows that the contours of pipe 1, the opening 4 in the 
pipe, the__box 5 and the lamp 6 may each be represented 
by three pairs of segments 14, 15 and 16 (in this case 
called limbs) that are parallel or approximately 
parallel, an ellipse 17, nine straight line segments 
18, and a dot 19. In practice, these contours separate 
portions with different colors on the image and are 
therefore discontinuities, which are used to measure 
them; this is the task performed by the positioning 
module 20. 

Positioning of natural contours on an image is 
based on ,! def ormable 11 models or active contours (see 
the article by Kaas, Witkin and Terzopoulos "Snake: 
active contour models 11 published in the International 
Journal of Computer Vision, 1(4), p 321 to 331, January 
1988 and the Bascle's thesis at the University of Nice 
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Sophia Antipolis (January 1994) "Contributions et 
applications des model es deformables en vision par 
ordinateur - Contributions and applications of 
deformable models in computer vision" . They consist of 
digitally varying a deformable contour model starting 
from an initial position while calculating its energy 
after each deformation. This energy conventionally 
includes two terms, the first of which expresses the 
geometric regularity of the model and takes account of 
any physical properties, and the second' takes : -account 
of. the ".match .between the model and the experimental 
image obtained. .: v Specifically, "the .purposed .. of . this 
processing :*.±s,ytox regularize the model: by. -'reducing its 
local irregularities, usually due to noise', without 
getting too. different from the information in the 
image; but it only works well on fairly simple images, 
which is not the case here. „ Furthermore, this 
invention proposes an improvement by describing some 
elements of the image by global geometric parameters. 
Therefore, we can say that the environment models that 
will be obtained will be both deformable and 
parametric . 

The shapes of the contours in which we are 
interested here are simple and belong to a few 
preferred types that are encountered very frequently in 
reality; as shown on figure IB, the main types are a 
dot, straight line, double straight line and an 
ellipse. A dot will be modeled by its Cartesian 
coordinates x and y on the image. Since the images are 
finite, the straight lines will be modeled by segments 
and they will have four parameters on the image, namely 
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the coordinates x 0 and y 0 of the middle of the segment, 
the length L of the segment and the angle 0 formed 
between the segment and the horizontal on the image, as 
shown on figure 3 . All dots on the segment will 
satisfy the following equations: 



(1) 



X = X n + U — COS0 

0 2 



y = y 0 +u — sine 



where u is -between -1 and +1 and 0 is between: 0 and n . 

Vj- A^.cylinder . will be def ined by .c: ifes^Gontours or 
limbsp. It l ..will consist of - two;;v>par.al;fel segments, 
unless ; r.::the : ' ■■ perspective effect-- as: ; -considered. A 
suitable model is shown in figure 4 vthat comprises five 
parameters; r the coordinates x 0 and y 0 of the " isocenter 
of gravity of; the system formed by the * two .*, segments , 
the common length _L .of the two segments, the angle 0 
formed between the two segments and the horizontal and 
the distance d separating the two segments. The 
coordinates of the dots of the two segments are given 
by the following equations: 



(2) 



x = x 0 + u y cos (0)+ ^-sin (0) 

y = y 0 + u ^ sin ( 6 )-| c °s(e) 



and ( 3 ) 



x = x 0 +u^cos(0)-^sin(0) 
y = y 0 +u^sin(0)+^cos(0) 



where u (path parameter) is between -1 and +1. 
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But if we want to take account of a perspective 
effect, the previous model can be enriched by 
parameters 50 expressing deviations in opposite 
directions and making the two segments converge, as 
shown in figure 5; there are then four dots selected 
by the operator. The equations 



(4) 



x = x 0 +u^cos(e + S6) +| sin (e) 
y = y 0 +u^sin (0 -h89)-^cos(e) 



and (5) 



x = x 0 + u y cos (0 - S0) 1 - ~ sin (0) 
y = y 0 + u ~ sin (0 -.- 8ft) j-JL cos (0) 



replace equations 2 and 3. 

10 Projection of a circle in space onto a two- 

dimensional image forms an ellipse, and figure 6 shows 
one •■ possible way of defining parameters- for this 
ellipse; the parameters are the coordinates x 0 and y 0 
of the center, the lengths l x and 1 2 of the large and 

15 small axis and the orientation 9 of the large axis with 
respect to the horizontal. The equations 

\x = x 0 +l t cos(0)cos(u)- 1 2 sin(0)sin(u) 
^x = y 0 -hi j sin(0)cos(u)-l 2 cos(0)cos(u) 

give the coordinates of dots on the ellipse, where u is 

a curved abscissa parameter between 0 and 2n . 

20 The process begins by initializing the 

representation of the environment, usually manually, in 
which an operator examines one of the images on a 
computer screen and marks the contours to be modeled. 
After choosing the appropriate contour type, he chooses 

25 a sufficient number of dots on the screen to define 



(6) 
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this contour and enable a first calculation of the 
parameters . 

These dots are marked by stars on figures 3 and 4; 
they are the ends of the segment, and the ends of one 
5 limb of the cylinder and a dot on the other limb. An 
ellipse is defined by 5 dots. 

The next step is to match the contour selected by 
the operator, or selected automatically on the image by 
using a potential function using calculations made by 
10 the positioning module 20. In general, an improvement" 
fV . ' po^f- - a ' model on an image. h is 'evaluated by successive : 
\ '.' .- reduction of a function ;. .Pe : ^called the potential, 
^^function that includes severaiftexms. In most cases,' 
the energy term alone is -sufficient . The image is 
15 processed by calculating the differences in digitized 
shades of gray of adjacent dots/, to relate a high 
potential intensity to each dot on the image if the dot 
is within an area with a uniform color, and a low 
potential intensity if it is located in a transition or 

2 0 color discontinuity area. This is done for each dot on 

the image. If a potential image was shown, it would 
show dark areas around the contours of objects, and 
usually a light background elsewhere. The sum of the 
potential of a contour is calculated on all its dots, 
25 and then a digital analysis algorithm by reduced 
gradient is used to calculate potential variations as a 
function of the variation of contour parameters. In 
this case, the objective is to minimize the root mean 
square e of the potential Pe along the contour C, using 

3 0 the following equation 
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(7) e m in(a) 



= £ p e( x ( a )>y( a )) 



x,yeC, 



where a is the model parameters vector and x, y are the 
abscissas and ordinates of the dots on the contour. 
Apart from the rate of convergence, this digital tool 
has the advantage that it provides an evaluation of the 
covariance matrix on the estimated model, denoted A a . 
This information will be used by the three-dimensional 
reconstruction and positioning module. 

A special distance given. by equation p :.. 



being quadratic close to zero, in other words to the 
contour, and approximately constant when the Euclidian 
distance between dots on the image d becomes large. a 
is a fixed coefficient. This distance is comparable to 
a weighting coefficient that attenuates the influence 
of remote dots in the calculation of the potential Pe . 

However, an additional potential term is used in 
addition to the previous term Pe for cylinder contours. 
It frequently arises that these elements are affected 
by lighting variations that create highly reflecting 
bands of brightness towards which the deformable model 
may converge by confusing them with contours. The use 
of this additional term avoids this danger; it is a 
conventionally very high potential term for strongly 
illuminated dots; the total potential thus modified 
becomes high close to reflecting bands, which pushes 




is used to calculate the potential Pe of dots on the 
image. This special distance has the advantages of 
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the modeled contours towards real contours of the 
cylinder. 

Note also the influence of geometric aberrations 
introduced by the lenses of an objective; a straight 
5 line in space is projected onto the image as a curved 
segment, rather than a straight line segment- The 
deformable models described here cannot give a perfect 
approximation of this type of deformed parts, but a 
process for correction of geometric aberrations can be 
. 10 used to apply the process according to the invention to 
M* corrected images, obtained without distortion. This. 

. •• correction process is made for all dots on the: -..image at. 

f?' v "' K *y: the same time in advance, and the corrected image. cis: 

"OCT stored in memory 12 . • 

l» 15 Geometric aberrations are composed of two terms,: 



5' 



including one radial distortion term that moves a dot- 
radially with respect to the optical center of the 
image and is expressed as a polynomial with equation 



=3 E 

m 

Q (9) 5 r (r) = Kn.r 3 + K 2 r 5 + K 3 r 7 

fcft 



(10) 



20 as a function of the radial distance r = -^x 2 + y 2 ; and 
a tangential distortion term that includes a tangential 
component and a radial component in accordance with the 
following equations : 

[S T (x) = P, (r 2 + 2x) + 2P 2 xy 
l8 T (x) = P 2 (r 2 -f2y) + 2P l xy 
25 The coefficients K x# K 2 , K 3 and Pi and P 2 are 

distortion coefficients estimated while the camera is 
being calibrated. 

The radial distortion is estimated by a 
preliminary calculation of an aberration table as a 
30 function of the radial distance. For each radial 
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distance r D from the center of a distorted calibration 
image, this table contains the corresponding distance 
r m of the same position in the undistorted image. The 
separation between successive values of the distances 
r D stored in the table is chosen such that the minimum 
precision A between the successive values of the 
corrected distance r^ is respected. The precision of 
this process can be as high as one tenth of the 
distance between two successive dots on the image. 

-It is not intended to use the same method "in .this 
invention- to take 7 "account of tangential. \; distortion, 
since tables* giving corrections as a f dnctaofH oSihe." x 
and y coordinates :;: 'should apply to all dots ioh-:.the r - image 
and would occupy too much space in memory. This is why 
it is recommended that an equation roots search 
algorithm based on equations (10) should be used, 'such 
as Powell's algorithm that- is well known to a person 
skilled in the art, if these tangential distortions 
have to be taken into account . 

We will now go on to describe the second module 21 
of the operating system, which is a module for 
reconstruction and positioning that makes use of the 
positions of contours of objects detected previously on 
the images to determine the position of these objects 
in the environment, in other words to build up a three- 
dimensional representation of the environment while 
calculating the position of the image sensor 7 in a 
positioning step. The process is recurrent, in other 
words the images are used in sequence, the 
representation of the environment being added to and 
corrected each time to make it more precise. It is an 
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application of the Kalman filter. This presentation 
describes the use of a stereoscopic sensor 7 with two 
cameras , but the process would be applicable to a 
sensor with a single camera; reconstruction and 

positioning can be evaluated except for a scale factor, 
that can be determined by inputting additional 
information into the system, such as a distance between 
two dots or the radius of a cylinder. 

The following describes the formulas that relate 
vthe. vector, x k of parameters of ? the .object ;detected in 

an absolute'' 'coordinate system ' and the vebtior 1 ./ of its 
observation ^ coordinates in this. - image ' f -;b -for /.".a camera 
with index ^ i 1 of the sensor that feook-V-an image at 
instant k'. The position of the camera will be noted by 

a rotation matrix R' k and a translation ' vector; 4 in the 
absolute coordinate system. Transfer ' formulas are 
denoted by th~e~ letter h. 

In the case of a dot, the equations 







u-f ^ 




(11) 


hj,(x k ,z k ) = 




= 0 














I Z k J 




(x k ,y k/ z k ) fc 


= R' k (x, 


y, z) fc 


+ t k 



where x k = (x, y, z) fc , z ! k =(u, v) . 

In the case of a straight line, x k and z' k are 

defined by vectors (13) x k = (x, y, z, p, cp) t , z k =(x, y, 

z, P/ <p) t , z , k =(u / v, 9), in which P and (p are the 

spherical coordinates of the unit vector of the 

straight line and 0 is the angle formed by its 
projection onto the image; the formulas 
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((m, - m p )xv, N \ 
V lKxV k ) J 

where x is the vector product, define the conditions to 
be satisfied, in which (m k , v k ) are the parameters of 
the straight line (the coordinates of one of its dots 
m k and its unit vector) in accordance with the 
following equations : 

(15) m k = R k m + t 1 k? v k = R fc v , 
mp represents the coordinates of the projection of dot 
m k onto ' the image, m T is the middle " of the segment 
"'detected" on the image and v x is t the^urilt . vector of the 
•de*gmfent :: in ' accordance with f±<^u^ 7-y^:axxd m x and v z are 
deduced from z k . " *; * ■ 

An infinite cylinder is defined by the vector 

(16) x k = (x, y, z, p, <p, T) t t 

in which x, "y and z are the coordinates (denoted m) of 

a dot on its axis, p and cp are the spherical 

coordinates (denoted v) of the unit vector along its 

axis, and r is its radius. The equations 

(17) m k = R k m + t k and v k = R l k v 

express the position of the axis of the cylinder in the 
coordinate system of camera i at time k . The 
coordinates of its limbs (m x , v x ) and (m 2 , v 2 ) , and mp x 
and mp 2 , the projections of dots m x and m 2 of the limbs 
onto the image, are also calculated. The measured 
parameters on the image 

(18) (u, v, 0, 50, d) 
are used to deduce the observation vector z k = (ui,v lf 
@i/ v 2 , 0 2 ) corresponding to the mid-dots and the 
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orientations of the two observed limbs and the 
following measurement equation is obtained: 

. . v n .(m T xv.) 

(19) k[ y (x k ,z[)= uK 1 lJ = 0 

m n -m p2 )xv I2 

^v I2 .(m 2 xv 2 j 

Figure 8 shows these parameters. vn and mn, v I2 
and m I2 are deduced from z k , as in the case of the 
straight line. 

" . The circle is defined by a.state vector conform 
with the following f ormula 4 : . . ;;. .«•■... ci". . 

\.y. ; )j, : -i (20) x k = (x; yy yftr: .9/ r ) t / < 

- where x, y and z denote- 1 the coordinates of its 
center, P and cp the spherical coordinates of the unit 
vector along its normal and r. is its radius. 
Furthermore, the formulas 

(21) m k = RJ.rn-ftJ. and v k = RJ.V 
are applicable. If observation coordinates are 

represented by the function 

(22) z' k =(u ? v 5 l 1? l 2? 0) , 
the following equations 



(23) h;(x k; z' k ) = 



= 0 



'q 0 -((b 2 (x 2 + y 2 + z 2 -r 2 ) + l-2by k )/Q) 
q, -((2ab(x 2 +yl+z 2 k -r 2 2bx k -2ay k )/Q)) 
q 2 -((2ac(x 2 + y 2 + z 2 -r 2 )-2cx k -2az k )/Q 
q 3 -((2bc(x 2 +y 2 k + z 2 -r 2 )-2cy k -2bz k )/Q 
^q 4 -((c 2 (x 2 + y 2 +z 2 - r 2 ) + l-2cz k )/Q) j 

where Q = a 2 ( x k + y k + z k - r 2 ) + 1 -2bx k express the transfer 
between the state vector and observations, in which 
qo/ — , q4 are derived from conversion of parameters (22) 
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to obtain a representation of the ellipse in implicit 
form such that u 2 + q 0 v 2 + qiuv + q 2 + q 3 v + q 4 = 0 . 

We will now go on to the description of the 
reconstruction process in the special case of a sensor 
formed from two cameras fixed with respect to each 
other, denoted by their indexes 1 and r and 
simultaneously taking an image. For a dot, the global 
observation vector can be expressed by 

(24) z k = (u 1 ,v 1 / u r ,v r ,Xk/Pk/a k/ t xk , t yk/ t zk ) 
"where u 1 , v 1 , u r and v r are- the coordinates of the dot * 
on the two images and . the . -other parameters - are - the 
^orientation and translation;-- vectors of the sensor ; in 
the. absolute coordinate system. - The dot observation 
function is then given by the .following equation \ 



(25) hp (x k , z k ) != 



= 0 , 



P 

for which the solution (which is a duplication of 
equation (11) for the two cameras) gives an evaluation 
of the state vector x k of the dot, composed of 
coordinates x, y and z in the absolute coordinate 
system . 

The position of a straight line is determined by 
obtaining an observation vector 

(26) z k = (u 1 ,v 1 ,e 1 ,u r / v r ,e r ,Xk,Pk,a k , t xk , t yk/ t zk ) t 
and solving the following equations 



(27) h d (x k , z k ) = 



= 0, 



analogically; note that the 9 parameters are the 
angles between the projections of the straight line 
onto the images 1 and r and the horizontal. However, 
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note that since straight line segments are observed 
rather than the straight lines themselves, the state 
vector for a straight line is given by the formula 

(28) x k = (a, b, p, q) t , 
5 rather than by the coordinates of a dot on the straight 
line and the unit vector along this straight line. For 
each acquisition, the straight line estimated by the 
parameters of the state vector a, b, p and q is 
expressed in the form of a finite straight line with 

10 parameters x, y, z, P,' q> -and 1 where 1' denotes the 
length of the segment -v and the coordinates x , r^va-n'dv zv 
denote the .middle? of;- this, segment. These . coordinates ; 
x, y and z are evaluated by reprojection into "ithe 
image. ■ The definition of parameters a, b, p and~q is: 

15 as follows: 

- the straight line has a unit vector (1, a, b) 

- - and a position vector (0-,- p, q) unless it is 

perpendicular to the 0 X axis; 

- it may be defined by the unit vector (a, 1, b) 
20 and a position vector (p, 0, q) unless it is 

perpendicular to the O y axis; 

- and by a unit vector (n, b, 1) and a position 
vector (p, q, 0) , unless it is perpendicular to 
the O z axis. A preferred convention defines a 

25 priority when several of these representations 

are possible. 

The cylinder is also defined in the representation 
by the parameters a, b, p and q of its axis and by its 
radius, using the formula 
30 (29) x k = (a, b, p, q, r) t . 

The observation vector is defined by the formula 
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(30) z k =(u! s v},e; 5 ui,v^ei ^[^[^e^u^v^e^^p.a^t.^ty^t^) 1 . 

The system of equations 



(31) h cy (x k/ z k ) = 



^(x k ,zL)^ 



= 0 



^h cy (x k ,z k ) ^ 

must be solved. Finally, the state vector of a circle 
is defined by the following formula 

(32) x k = (x, y, z, p, cp, r)*, 
and the observation vector is defined by the formula 

(33) zk =• (uS-vVlj,-^ /0 1 ,u r / v r / l[J r 2 / 9 r / a kV p k/ Xic',t xk/ t yl ^, t z]c ) t / 
and the syst t em f ..'of equations • * , .* r\ 



10. .. . (34:) r h cy (x k , z k ) = 



N must be solved. 



h C y ( X k ' Z k 



= 0 



e The estimated position of the object is refined 

U 

Lj for each new acquisition. When an object appears in a 

FU pair of images for the first time, this estimate is 

m 

g| 15 initialized by a preliminary reconstruction by 

N* triangulation . Prior art already contains descriptions 

of such methods. A suitable initialization makes the 
estimate of the position of the object converge more 
quickly for each new image. 
20 Reconstruction of the three-dimensional 

environment requires the position of the sensor to be 
determined; this position is usually not known, or is 
known but with an insufficient precision. For each new 
acquisition, dots previously reconstructed in the 
25 environment are used and their observation vector is 
used for pre-positioning of the sensor by searching for 

(36) min (Xk, Pk, a k , t x , t y , t z ) =£||h J p (x k ,z J k "~ 

J 
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q 



2 f 



in other words the values (3 k , a k , t xk , t yk/ t zk that 

give the best agreement between the representation of 
the environment and its image on the cameras (h close 
to 0) for all dots j in the model. The following 
5 equations are then solved recurrently: 

(37) h p (x k/ z k ) = 0, h d (x k/ z k ) = 0, h cy (x k ,z k ) = 0, 
or h c (x k , z k ) =0 

(one for each object already built, depending on the 
category of the object), in which observation vectors 
10 z k are given by the appropriate formula" ■ . '•" 

; %s\ vrc(.38)- z k = (u 1 , v 1 , u r , v^ , x , y , z , ,) fc > -0 : 

v.- ■ 'V:. : zW =- (u\v\e\u r ,v r , e r ,x,y,z,p',<p) x , . • 

^ z k = (ul^ : vj; 6 ; , u^, v*, 0 uf , v[ , 6 ; , u^, v^, 6 r 2 , x, y, z,p, : ^'r) 1 . 

N or z k = (u\v\ i; ? l» / 9 1 / u r / v r / l[ ? l^ / 9 r / x ? y ? z ? p,(p ? r ) t 

s 15 this is another application of the Kalman f.ilter in 

which the estimated state vector in this case is (Xk, 
Pk, cx k/ t xk , t yk/ t zk ) . Module 22 performs this 

positioning . 

The identification module 23 of the system 
20 automatically identifies at least some of the contours 
defined in the previous calculations, each time that an 
image is taken. It is proposed to proceed as follows: 

- select a previous image k 0 , preferably close to 
the current image k concerning -positions and 

25 orientations of the photo; 

- select points of interest I 0 on this previous 
image k 0 , which can be done automatically, the 
points of interest having the general property 
that the brightness gradient close to them is 

30 high, and is not usually sensitive to changes in 



m 
m 
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image taking conditions (lighting, zoom, view 
exposure) . Therefore a characteristic dot 

already identified with an image k 0 will usually 
be identified again on the next image k, unless 
5 it is hidden by another object in the 

environment ; 

- when the points of interest I 0 and I K in the two 
images have been found, they are made to 
correspond from one image to the next; this can 

10 ^be done using the brightness information close 

Lj f 

to -.each of them, since this is } ; wha t could best 

q ........ ...... 

O ./characterize them; it is codedr-im vector form 

us,ing different filters. For;-' each point of 
interest I k in the new image k> the module 

M 

jy* 15 searches among the points of interest I 0 in the 

*\ previous image k 0 to find the dot' most similar 

fy to it by calculating a correlation score or a 

m 

g?l vector distance (for example see the work done 

CP by C. Schmid " Appariement d ' images pa.r 

20 invariants loca.ux de niveaux de gris - Matching 

of images using local gray shade invariables 11 , 

INPG PhD thesis, 1996) ; 

- after correspondence has been identified between 
pairs of points of interest, assumed to 

25 originate from projections of a single dot in 

the environment onto two images, a 
correspondence matrix between the two images is 
thus obtained. It is then possible to use this 
matrix to project the previously estimated 

3 0 three-dimensional model onto the current image. 

The contours thus obtained are used for a 
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preliminary estimate of object contours for the 
new image k, and they are used by applying the 
process described above for module 2 0 to these 
contours, using deformable models. Therefore, 
5 the operator does not have to start selecting 

contours on the new image k all over again. 
Obviously, he can correct contours that appear 
to be incorrect or can eliminate contours that 
are hidden by other objects in the environment. 
,10 •.. ./.Since the program is . designed to eliminate 

' J? . • . v ■ ;.>V.contours hidden by qbj.ects -already included- in 

Cf • ;7 ; v-i-uthe model by itself, -the/ "operator should only. 

[fi F::' ./:-c;: need to eliminate hidden y -contours of objects 

03 / «■' - that have not yet been> identified. However, he 

M 

Id* 15 /. must introduce contours appearing on image k for 

"„ ' ' the first time. ' ':■ 

fU The last module performs a three-dimensional block_ 

m 

gPI calculation. This is done using module 24 when all 

Cf images in the environment have been used as described 

2 0 and a complete representation of the environment has 

been produced. The calculation is carried out as 

follows : 

- starting from parameters R' k and tj. known in 
advance for each image k, the projections of the 

25 contours of the representation onto the camera 

images planes are calculated; 

- the deviations between the positions of the 
projected contours and the positions of the same 
contours estimated previously on the same images 

30 are calculated; 
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- the positions of the contours in the 
representation are re-evaluated in order to 
minimize the deviations. 
The next step is to use a least squares method, 
5 minimizing a global error. A vector x = (xqi ... x Gn x M i 
...x MP ) T can be defined in which the x G values contain the 
parameters of all n objects of the representation and 
the x M values contain the parameters of the p photos 
(a, p, X' t x/ t y/ t z ) T , together with a measurement 
10 : - vector z that contains all observations made for each 
object and for each image . : -The adjustment made by: 
' ---module 24 is equivalent to minimizing an error function 
: v :: : F (x, z , a) in which a denotes known information about the 
image taking means (for example intrinsic parameters, 
15 "optical center, focal length, scale and distortion 
factors) or about the representation (for example the 
parameters of vector x that are assumed to be well 
determined or known). Weightings of the different 
parameters may be introduced. Therefore, this module 
20 24 can evaluate uncertainties of the representation of 
the environment and can reduce them by modifying 
estimated image taking parameters. 

Some parameters can be corrected or blocked. The 
parameters used are u and v for a dot, 0 and d 
25 (distance to the origin of the image coordinate system) 
for a straight line and each cylinder limb. 
Furthermore, the coordinates u and v of the ends of 
straight line and cylinder segments are also used. 

The block calculation can also be used to measure 
30 the position and orientation of one or several objects 
using a single image and a camera. This can only be 
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done if additional information about the objects is 
available; the geometric characteristics of each 

object must be known and injected into the block 
calculation. The measurement of projections of these 
said characteristics in a single image is sufficient to 
determine the position and orientation of the object. 
It will be necessary to make sure that a sufficient 
number of characteristics is available to evaluate all 
position and orientation parameters. 
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